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I.  INTRODUCTION 


In  October  2002,  the  Research,  Development,  and  Engineering  Command  (RDECOM)  was 
established  by  the  Army  Materiel  Command  (AMC)  to  integrate  the  research,  development,  and 
engineering  components  of  AMC  subordinate  Commands.  A  Virtual  Distributed  Lab  for 
Modeling  and  Simulation  (VDLMS)  was  initiated  and  selected  to  execute  the  RDE  Command's 
First  Application  (IstApp).  The  objectives  of  IstApp  were  to  provide  insights  into  the 
Networked  Fires  process  and  performance  for  Future  Combat  Systems  (FCS),  and  to  define  the 
baseline  capability  of  the  VDLMS  as  it  transitions  towards  the  Modeling  Architecture  for 
Technology  and  Research  Experimentation  (MATREX). 

IstApp  was  a  geographically  distributed  experiment.  The  innermost  tier  of  operation  was  a 
co-located  Distributed  Interactive  Simulation  (DIS)  network  at  a  single  site.  The  next  tier  was  a 
co-located  High  Level  Architecture  (HLA)  network  bridged  to  DIS  across  a  gateway.  The  outer 
tier  was  the  distributed  HLA  connectivity  to  the  other  host  sites.  This  approach  ensured  that 
connectivity  or  network  performance  roadblocks  in  outer  tiers  did  not  preclude  execution  of 
some  major  portion  of  the  architecture  to  produce  useful  analytic  results. 

Geographic  distribution  of  the  event  was  accomplished  by  linking  four  simulation  sites  with 
one  additional  Wide  Area  Network  (WAN)  monitoring  and  collaboration  server  site,  and 
physically  bringing  resources  from  the  other  VDLMS  organizations  to  the  four  simulation  sites, 
as  follows: 

A.  WAN  Monitoring  and  Collaboration  Server  Site 

Defense  Research  and  Engineering  Network  (DREN),  Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 

B.  Distributed  Simulation  Sites 

Aviation  and  Missile  Research,  Development,  and  Engineering  Center  (AMRDEC), 
Redstone  Arsenal,  AL. 

RDECOM  Simulation  Technology  Center  (STC),  Orlando,  FL 
Communications  and  Electronics  RDEC,  (NVESD,  CERDEC),  Ft.  Belvoir,  VA 
Redstone  Technical  Test  Center  (RTTC),  Redstone  Arsenal,  AL 

C.  Remote  Site  Organizations 

Armaments  RDEC  (ARDEC)  -  (at  AMRDEC) 

Army  Research  Lab  (ARL)  -  (at  Orlando) 

Tank  and  Automotive  RDEC  (T ARDEC)  -  (at  RTTC) 

CERDEC  Monmouth  -  (at  Redstone  and  Ft  Belvoir) 
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Depth  and  Simultaneous  Attack  Battle  Lab  (D&SA  BL)  -  (at  AMRDEC) 


A  more  indepth  view  of  the  RDE  IstApp  from  a  simulation  “lessons  learned” 
perspective,  can  be  found  in  Reference  1 . 

II.  RDE  ISTAPP  NETWORK  DESIGN  APPROACH 

The  design  approach  for  the  RDE  IstApp  was  effected  by  the  RDECOM  Chief  Architect, 
Mr.  Max  Lorenzo,  and  a  team  comprised  of  members  from  the  simulation,  network,  and  security 
communities  at  each  site,  the  DREN,  (to  include  technical  support  from  WareOnEarth 
Communications  Inc.  and  WorldCom/MCI),  and  also  technical  support  from  Cisco  and  Marconi. 
To  begin  the  design  process,  network  requirements  were  identified  and  design  approach 
established. 

A.  Network  Design  Requirements 

The  simulation  requirements  mandated  that  HLA  traffic  be  supported  on  the  backbone, 
and  that  DIS  traffic  be  contained  locally  at  a  site.  Thus,  multicast  and  TCP/IP  traffic  needed  to 
be  supported  on  the  backbone  and  User  Design  Protocol  (UDP)  broadcast  traffic  contained 
locally. 


Performance  characteristics  were  derived  based  on  previous  experiments  [2]  and 
projected  expectations.  It  was  determined  that  latency  from  site  to  site  should  not  exceed 
40  milliseconds,  and  that  local  jitter  should  be  less  than  1  millisecond.  Also,  a  guaranteed 
bandwidth  of  20  Mb/s  was  sought  for  the  simulation  sites  that  could  support  this  requirement. 

Security  was  also  an  important  design  consideration,  and  the  decision  was  made  to 
encrypt  the  backbone  of  the  network. 

B.  Network  Architecture  Design  Approach 

There  were  basically  three  main  components  to  be  considered  in  the  design  approach 
of  the  network  used  to  support  the  RDE  IstApp:  (1)  The  Local  Area  Network  (LAN),  which 
can  be  defined  as  the  network  equipment  that  provides  support  for  a  site’s  local  system 
infrastructure,  (2)  The  Metropolitan  Area  Network  (MAN),  which  can  be  defined  as  network 
equipment  that  resides  between  the  LAN  and  the  DREN  Service  Delivery  Point  (SDP),  and 
(3)  The  Wide  Area  Network  (WAN)  is  defined  as  those  devices  that  provide  networking  services 
connecting  the  distributed  sites.  The  DREN  was  used  to  provide  WAN  services  during  the  RDE 
IstApp. 
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1 .  Local  Area  Networks  (LAN) 

Early  in  the  design  process,  a  generic  LAN  design  was  proposed.  The  design  was 
established  to  allow  flexibility  so  as  to  support  Asynchronous  Transfer  Mode  (ATM)  and 
Ethernet  connectivity.  The  utilization  of  two  ATM  switches,  besides  permitting  ATM  local 
services,  also  afforded  a  more  robust  opportunity  to  measure  performance. 
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Figure  1.  Proposed  LAN  Design 

The  basic  LAN  designs  as  implemented  by  the  sites  are  as  follows: 

a.  AMRDEC 

This  site  provided  support  for  around  90  systems  and  several  printers.  A 
new  cable  plant,  enabled  to  support  classified  processing,  was  designed  and  installed  on  three 
floors  to  support  this  application.  The  Cisco  3500  switches  on  each  floor  supported  the 
FastEthemet  adapters  on  the  systems.  The  switches  were  trunked  with  lGb/s  connections.  At 
this  site  only  one  ATM  switch  was  supported.  The  Fastlane  was  connected  to  a  Cisco  5509 
switch.  A  standalone  DIS  network  was  installed.  A  gateway  on  this  network  connected  it  to  the 
HLA  network.  The  HLA  network  was  connected  to  the  DREN  via  an  OC3  connection. 
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b.  RTTC 


RTTC  supported  two  systems  on  the  HLA  network  and  one  system  on  the 
DIS  network.  A  Cisco  7204  (and  an  enhanced  ATM  adapter)  was  used  to  support  the  RDE 
IstApp  LAN.  A  secondary  ATM  switch  was  not  available.  An  OC3  connection  to  the  DREN 
was  utilized. 

c.  ARL 

Aberdeen  Proving  Ground  (ARL-APG)  hosted  the  FCS  Advanced 
Collaborative  Environment  (ACE)  system.  A  Cisco  7505  was  used  to  support  the  LAN 
requirements.  Marconi  ASX-200BX  ATM  switches  on  both  the  encrypted  and  unencrypted 
sides  of  the  Fastlane  were  installed.  The  Fastlane  provided  OC-3  ATM  encryption.  The  OC-12 
ARL-APG  site  is  connected  to  the  DREN.  Unclassified  lab  systems  were  a  Personal  Computer 
(PC)  for  e-mail,  and  an  SGI  Octane  for  testing  as  needed. 

d.  NVESD 

Night  Vision  configured  three  separate  networks  to  keep  the  broadcast 
traffic  down  to  a  minimum.  The  Situation  Awareness  traffic  was  separated  from  the  DIS 
Simulation  network  and  propagated  through  an  SA  Server.  The  HLA  traffic  was  translated  into 
DIS  for  the  simulation.  About  13  systems  were  supported  on  the  DIS  network,  and  3  systems 
supported  on  2  independent  HLA  networks.  A  Cisco  5509  was  used  to  support  the  HLA 
network.  Two  ATM  switches  were  available  for  the  support  of  this  application. 

e.  STC 

STC  configured  only  six  systems  to  communicate  both  HLA  and  DIS  via  a 
Cisco  3725  router.  The  DIS  traffic  was  isolated  using  a  single  hub,  and  a  MAK  gateway.  The 
traffic  was  translated  from  Ethemet/IP  to  ATM  using  a  Cisco  3725  router.  Only  native  HLA 
machines  and  the  HLA  network  interface  of  the  MAK  gateway  was  connected  to  the  Fast 
Ethernet  ports  on  the  router.  This  Fiber  Interface  was  then  connected  using  single  mode  fiber 
cables  to  the  single  mode  OC3  Fiber  Interface  on  the  KG-75  Fastlane.  The  outgoing  data  from 
the  Fastlane  was  encrypted  and  then  sent  out  on  another  OC3  Fiber  Interface  to  the  LE-155 
Marconi.  And  from  there,  the  data  was  sent  out  to  classified  WAN  over  the  DREN. 

2  MAN/  WAN  (DREN) 

The  DREN  was  chosen  to  provide  WAN  services.  The  DREN  is  a  sophisticated, 
robust  Department  of  Defense  (DoD)  communications  network  that  incorporates  the  best 
operational  capabilities  of  both  the  DoD  and  the  commercial  telecommunications  infrastructure. 
DREN  is  DoD’s  premier  long-haul  communication  service  provider  for  the  High  Performing 
Computing  (HPC)  community. 

The  DREN  provides  interoperable  ATM  and  Information  Processor  (IP)  services 
for  video,  audio,  imaging,  and  digital  data,  and  connects  to  other  research  and  academic 
networks  at  Next  Generation  Internet  Exchanges  and  GigaPops.  The  network  links  customer 
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sites  to  DoD’s  four  Major  Shared  Resource  Centers  (MSRC)  and  17  Distributed  Centers  (DC). 
The  DREN  has  over  70  IP  sites  with  over  35  of  these  sites  configured  with  additional  ATM 
functionality  to  support  AT-based  encryption  and  applications.  The  current  network  provides 
Digital  Service  3  (DS-3)  through  OC-48  connectivity. 

Wide  Area  connectivity  is  provided  by  Layer-2  equivalent  transport  between 
protected  SDP,  utilizing  a  combination  of  Multiprotocol  Label  Switching  (MPLS)  tunnels  and 
MPLS  Label  Switch  Paths  (LSP). 

An  ATM  service  is  supported  by  the  use  of  cell-relay  tunnels  across  LSPs.  DREN 
Core  Nodes  (DCN),  another  variation  of  the  DREN  SDP/PE  router,  are  located  at  selected 
MCI/WorldCom  vBNS+  network  nodes.  User  ATM  service  interfaces  are  connected  each 
through  the  cell-relay  tunnels  to  a  DCN,  which  provides  a  User  Network  Interface  (UNI)  service 
interface.  The  DCNs  are  fully  meshed  through  LSPs,  providing  a  Layer-2  equivalent  transport 
"cloud"  to  all  ATM  service  interfaces. 

The  decision  was  made  to  utilize  a  point-to-point  spoke  configuration  rather  than  a 
full  mesh  because  AMRDEC  was  centrally  located  and  a  focal  point  of  the  experiment;  i.e.,  data 
was  only  required  to  be  passed  back  and  forth  between  each  site  and  AMRDEC  (the  RTIexec 
was  executed  at  this  site),  a  spoke  configuration  significantly  reduced  the  complexity  of 
establishing  connectivity  to  each  site  and  a  spoke  configuration  significantly  reduced  the  amount 
of  time  spent  troubleshooting  a  faulty  data  path. 

A  spoke  configuration  reduced  the  overall  bandwidth  requirement  at  each  site’s 
gate  to  the  DREN  and  within  each  site’s  local  infrastructure.  A  spoke  configuration  also 
provided  AMRDEC  with  the  capability  to  enable  or  disable  Soft  Permanent  Virtual  Circuits 
(SPVCs)  to  each  site  as  necessary,  thus  conserving  bandwidth  when  connectivity  to  a  site  was  no 
longer  required. 

The  DREN  does  have  the  capability  to  utilize  SPVC  throughout  its  core.  Thus, 
use  of  Switched  Virtual  Circuits  (SVCs)  and/or  SPVCs  throughout  the  DREN  Core  is  mandatory 
unless  there  are  extenuating  circumstances  and  all  efforts  to  establish  connectivity  via  SVCs  or 
SPVCs  have  been  exhausted.  The  DREN  Program  Manager  must  approve  any  deviation  from 
this  policy. 


During  the  initial  planning  stages  of  this  experiment,  the  decision  was  made  to 
establish  connectivity  to  each  site  via  SPVCs  and  utilize  bandwidth  reservation  through  the  use 
of  Quality  of  Service  (QoS)  parameters  settings  provided  by  Marconi  ATM  Switch  products 
within  each  site’s  local  infrastructure.  Also,  it  was  decided  to  use  Fastlanes  to  establish  secure 
communications,  by  encrypting  the  ATM  backbone,  for  this  application. 

Each  SPVC  was  to  be  initiated  by  the  AMRDEC.  AMRDEC  would  establish  an 
SPVC  from  the  port  on  the  Black  ATM  switch  that  was  directly  connected  to  the  AMRDEC 
Fastlane  cipher  text  jack  to  each  the  port  at  each  site’s  Black  ATM  switch  connected  to  the 
Fastlane  cipher  text  jack. 
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A  Permanent  Virtual  Circuit  (PVC)  would  be  created  to  carry  the  traffic  through 
the  Fastlane  to  the  edge  device  at  each  site.  The  end  device  at  each  site  consisted  of  a  Cisco 
product.  The  Cisco  products  utilized  did  not  have  the  capability  to  establish  SPVCs. 

The  decision  was  also  made  to  use  Request  For  Comment  (RFC)  2684  [3],  which 
is  “Multiprotocol  Encapsulation  over  ATM  Adaptation  Layer  5,”  on  the  edge.  RFC  2684  is  a 
replacement  for  RFC  1483,  and  describes  two  encapsulation  methods  for  carrying  network 
interconnect  traffic  over  AAL  type  5  over  ATM.  The  use  of  LANE  (LAN  Emulation)  was 
considered,  but  according  to  Cisco  and  Marconi  was  not  as  efficient  and  would  introduce 
unwanted  latency.  LANE  is  an  ATM  service  defined  by  the  ATM  Forum  specification  “LAN 
Emulation  over  ATM,”  ATM  FORUM  94-0035.  Classical  IP  (CLIP)  as  described  in  RFC 
1577,  or  “Classical  IP  and  ARP  over  ATM”  was  considered,  but  does  not  provide  support  for 
multicast  traffic 

To  further  simplify  the  structure  of  this  network  and  reduce  latency,  the  decision 
was  made  not  to  route.  One  IP  range  was  assigned  the  backbone,  and  the  traffic  was  bridged  on 
the  edge.  The  network  was  small  enough  that  this  was  possible.  Every  effort  was  made  to 
reduce  latency,  blend  the  MAN  into  the  WAN  layer,  minimize  the  number  of  interim  switches 
and  routers  in  the  path,  and  put  the  control  on  the  edge,  so  that  each  site  could  effectively 
manage  their  site.  This  also  served  to  simplify  troubleshooting  efforts,  as  the  focus  for  network 
management  was  well  defined. 
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III.  RDE  IstAPP  NETWORK  DESIGN  IMPLEMENTATION  AND  OBSERVATIONS 


After  the  network  design  approach  had  been  determined,  the  network  team  worked  to 
implement  and  test  the  design  at  each  site. 

A  Network  Architecture  Overview 

Figure  2  provides  a  high  level  overview  of  the  network  that  was  deployed  for  the  RDE 
IstApp.  Early  in  the  design  process,  a  configuration  chart  was  drawn  that  depicted  the  models 
and  operating  system  software  levels  of  all  the  main  ATM  and  Ethernet  switches  and  routers  to 
be  used  for  the  network.  The  equipment  vendors’  support  teams  were  contacted  to  provide 
advice  regarding  any  observable  issues. 
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Figure  2.  RDE  IstApp  Network  Overview 

B  Network  Design  Implementations  and  Observations 

During  the  implementation  phase,  there  were  many  challenges,  and  also  interesting 
observations,  that  were  noted  during  the  establishment  of  the  LAN  and  the  MAN/WAN. 

1 .  Network  Design  Implementations  and  Observations  -  LAN 

Some  site  network  design  implementations  and  observations  are  as  follows: 
a.  AMRDEC 

In  preparing  the  site  equipment,  the  IOS  on  the  Cisco  5509  was  upgraded, 
and  a  new  supervisor  module  was  purchased  for  the  existing  black  ATM  switch.  As  a  new  cable 
plant  had  been  installed  for  this  effort,  all  drops  and  cables  to  be  used  for  connecting  systems 
were  tested  for  continuity  prior  to  the  application.  During  the  application,  a  new  network 
performance  tool  called  “Observer”  was  used  to  monitor  the  health  and  status  of  the  LAN  (the 
DIS  and  HLA  networks). 
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Also  to  be  noted,  the  Redstone  DOIM  ATM  switch  needed  to  be  upgraded, 
though  there  were  no  resources  at  the  time  to  support  this.  This  was  problematic,  but  did  not 
have  a  detrimental  effect  on  the  exercise. 

b.  RTTC 

RTTC  upgraded  the  IOS  on  the  7204  router  and  also  borrowed  an  updated 
enhanced  ATM  adapter  to  replace  an  older  one  in  the  unit.  The  ATM  switch’s  operating  system 
was  also  upgraded. 


c.  ARL 

ARL-APG  hosted  the  FCS  program  ACE  software.  With  FCS  ACE, 
presentations,  applications,  audio,  and  video  can  be  shared  from  a  presenter  to  up  to  20  users. 

The  users  have  interactive  capabilities  using  audio,  whiteboard,  and  text  chat  tool  capabilities. 
The  user  presentations  at  Redstone  were  distributed  to  Orlando,  Aberdeen,  and  Ft.  Belvoir  via 
the  ACE  server  at  Aberdeen. 

Daily  testing  of  the  connection  showed  initial  packet  loss  of  0.2  percent 
from  the  ARL  site.  Originally,  the  SGI  02  network  test  host  was  directly  connected  to  the  Cisco 
7505  router  using  a  crossover  cable.  Once  the  ACE  client  PC  and  the  Ethernet  switch  were 
added,  the  packet  loss  in  testing  dropped  to  0.0  percent.  Ethernet  Duplex  settings  and  cable 
quality  can  be  factors  in  any  network  installation.  This  lesson  learned  led  ARL  to  implement  a 
Fast  Ethernet  switch  on  the  connection  to  the  FCS  ACE  server. 

d.  NVESD 

NVESD  had  introduced  the  use  of  the  multicast  conferencing  toolset,  to 
include  SDR  (VIC,  and  VAT,  etc.)  in  an  earlier  joint  exercise.  This  toolset  was  used  in  this 
environment  and  again  provided  multicast  performance  information  and  was  also  used  to  support 
presentation  feeds  by  the  FCS  ACE. 

There  seemed  to  be  a  slight  problem  with  entity  propagation  through  the 
MAK  gateway.  There  was  a  time  delay  and  not  all  entities  showed  up.  This  could  have  been 
due  to  the  TCP  stack  on  the  Windows  MAK  gateway  platform.  The  network  seemed  fairly  solid 
both  on  the  LAN  and  WAN. 

e.  STC 

STC  installed  a  new  Cisco  3725  in  support  of  the  RDE  IstApp.  A  Multi- 
Router  Traffic  Grapher  was  utilized  to  monitor  active  bandwidth  usage  from  each  of  the  routers, 
or  Layer  3  switches,  connected  to  the  WAN.  It  was  observed  that  simulation  latency  and 
anomalies  did  not  occur  due  to  network  traffic.  Bandwidth  never  reached  critical  levels  at  STC. 
Some  delays  of  entity  traffic  from  the  MAK  gateway  and  DIS  side  were  observed,  but  these 
anomalies  may  be  the  result  of  gateway  configuration  issues. 
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2.  Network  Design  Implementations  and  Observations  -  MAN/WAN 


Originally,  it  was  thought  that  a  tool  called  NETPerf  would  be  utilized  to  test 
performance  on  each  point-to-point  connection.  However,  as  the  implementation  of  the  overall 
network  design  was  performed,  the  results  provided  by  NETPerf  were  insufficient  to  assist  in 
troubleshooting  the  problems  encountered.  An  alternative  tool  called  NUTTCP[4]  was  utilized 
instead.  NUTTCP  is  a  throughput  test  that  is  used  to  send  TCP  or  UDP  packets  from  one  site  to 
another.  NUTTCP  3.6.1  was  utilized  to  validate  that  payload  data  transmitted  at  18  Mb/sec  could 
be  sustained  on  the  entire  connection  with  little  or  no  cell  or  packet  loss.  No  other  traffic  utilized 
the  SPYC  while  the  NUTTCP  tests  were  performed. 

Because  of  overhead  associated  with  ATM  cells  equals  roughly  9  percent,  to  send 
cells  at  approximately  20  Mb/sec,  NUTTCP  was  configured  for  a  maximum  of  18  Mb/sec,  send 
and  receive  UDP  traffic  with  a  payload  of  1440  bytes  over  a  10-second  period,  incrementing 
every  second. 


Both  receive  and  transmit  tests  were  performed  to  validate  whether  or  not  an 
18  Mb/sec  throughput  could  be  achieved  to  and  from  NVSED,  RTTC,  ARL,  and  AMRDEC. 

A  4  Mb/sec  throughput  factor  was  used  to  test  STC  to  AMRDEC. 

C.  Network  Design  Observations 

As  stated  before,  SPVCs  are  the  connection  of  choice  to  quickly  provide  reliable 
dedicated  point-to-point  connectivity  between  sites.  All  SPVCs  originated  at  the  AMRDEC. 

The  use  of  SPVCs,  as  originally  thought,  would  also  provide  the  capability  to  establish 
dedicated  bandwidth  reservation  from  AMRDEC  to  each  site.  Bandwidth  reservation  was  not  a 
critical  issue  through  the  DREN  cloud,  but  could  be  an  issue  on  each  site’s  LAN. 

The  establishment  of  QoS  between  the  AMRDEC  and  each  site  was  abandoned  due  to 
the  inability  of  the  utilized  hardware  at  Ft.  Belvoir,  AMRDEC,  and  RTTC  to  establish  a 
20Mb/sec  CBR  PVC  without  any  problems. 

First,  the  Cisco  ATM  modules  utilized  at  Ft.  Belvoir  and  RTTC  did  not  have  the 
capability  to  configure  a  CBR  PVC.  To  document  this  inability,  rate  shaping  of  a  VBRnrt  and  a 
UBR+  connection  was  tested  between  Ft.  Belvoir  and  the  AMRDEC.  A  series  of  tests  utilizing 
NUTTCP  and  ping  were  performed  to  note  the  problems  that  ensued. 

If  a  UPC  contract  is  applied  to  an  SPVC,  then  Traffic  Shaping  at  each  edge  device  is 
mandatory.  For  example,  if  a  UPC  contract  is  configured  for  CBR  and  a  22.5  Mb/sec  Peak  Cell 
Rate  enable  on  a  SPVC,  as  was  attempted  in  this  experiment,  and  there  is  no  rate  shaping  being 
performed  by  each  edge  device,  then  the  edge  device  would  attempt  to  send  data  at  the  line  rate 
of  its  ATM  interface  card. 

Thus,  if  a  site  is  utilizing  an  OC3  ATM  interface  with  no  rate  shaping,  data  would  be 
transmitted  to  the  SPVC  at  155  Mb/sec.  With  a  22.5  Mb/sec  Peak  Cell  Rate  setting  on  the  UPC 
contract,  the  data  would  be  policed  and,  therefore,  not  passed. 
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Thus,  in  order  for  traffic  not  to  be  policed,  edge  device  rate  shaping  must  be 
configured  so  that  bandwidth  parameters  are  slightly  below  the  PCR  of  a  CBR  UPC  contract. 

Since  bandwidth  was  not  an  issue,  the  decision  was  made  to  go  with  UBR  connections 
between  all  sites  in  the  RDE  IstApp. 

D.  RDE  IstApp  Network  Performance 

One  tool  that  was  provided  was  a  system  known  as  the  Active  Measurement  Program 
(AMP).  The  DREN  has  deployed  AMPs  [5]  at  many  of  its  major  nodes  over  the  past  three  years. 
AMP  was  created  by  the  National  Laboratory  for  Applied  Networking  Research  (NLANR) 
which  is  funded  by  the  National  Science  Foundation  to  instrument  and  measure  the  vBNS+  [6] 
and  Abilene  [7]  networks.  WareOnEarth  Communications  became  a  partner  of  NLANR  in  1999, 
and  has  deployed  AMP  systems  on  the  DREN  [8],  NIPRnet,  and  the  Navy  Marine  Corps  Intranet 
(NMCI)  [9].  ' 

AMP  collects  long-term  delay,  loss,  and  routing  information  between  a  full  mesh  of 
measurement  machines.  On  DREN,  this  is  done  primarily  by  sending  four  randomized  "pings" 
per  minute,  and  a  traceroute  every  10  minutes,  between  all  pairs  of  measurement  systems.  This 
data  is  downloaded  in  near  real-time  to  a  central  server,  which  can  then  analyze  the  data  and 
display  various  results  via  a  web  server  interface  [10].  The  DREN  AMP  systems  also  host 
numerous  network  testing  tools,  including  end  user  accessible  NUTTCP  servers,  and  normally 
run  a  mesh  of  treno  throughput  and  mping  [11]  load  tests  during  early  weekend  hours. 

For  the  RDE  IstApp,  the  decision  was  made  to  extend  the  DREN  AMP  system  to 
include  the  four  major  sites  involved.  ARL  already  had  an  AMP  system,  but  new  ones  were 
placed  at  Ft.  Belvoir,  Huntsville,  and  Orlando.  Because  RDE  would  be  running  almost  all  of  its 
traffic  over  ATM,  the  AMP  systems  were  equipped  with  both  Ethernet  and  ATM  interfaces. 

Unfortunately,  the  AMP  system  at  NVESD  was  not  installed  in  time  for  the  RDE 
IstApp,  and  the  system  at  AMRDEC/RTTC  had  unresolved  ATM  problems.  This  limited 
the  amount  of  data  the  AMP  system  could  collect  during  the  experiment.  It  is  hoped  that 
deployment  will  be  completed  in  the  future  to  provide  persistent  long-term  performance  data. 

An  example  of  AMP  data  from  the  experiment  is  shown  in  Figures  3  through  5. 

All  of  the  graphs  show  one  day  of  data  from  Wednesday,  23  April,  for  the  path  between 
AMRDEC/RTTC  and  STC.  The  first  shows  the  roundtrip  time,  which  was  very  stable  with  a 
mean  of  29.9  milliseconds  and  a  standard  deviation  of  0.31  milliseconds.  The  delay  distribution 
is  shown  in  the  second  graph.  The  jitter  was  below  one  millisecond  as  desired.  The  final  graph 
shows  packet  loss.  Of  the  5,760  probe  packets  (4  per  minute)  only  8  were  lost,  for  a  0. 14  percent 
loss  over  the  24-hour  period. 
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Figure  3.  AMP  Round  Trip  Time 
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Figure  4.  AMP  Round  Trip  Time  Distribution 
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Figure  5.  AMP  Loss  Data 


During  the  RDE  IstApp  “Runs  for  Record  (which  were  conducted  over  a  two-week 
period),”  throughput,  latency,  and  bandwidth  tests  were  performed.  Every  morning  each  site 
performed  ping  tests  to  each  of  the  other  sites  to  measure  latency.  Also,  NUTTCP  was 
performed  from  each  site  to  AMRDEC  to  test  throughput.  AMRDEC  monitored  bandwidth  with 
the  Observer  tool.  Figures  6  and  7  provide  some  representative  data  collected  during  the  “Runs 
for  Record.”  During  Week  1  of  the  “Runs  for  Record,”  a  three-hour  vignette  was  run  that 
consisted  of  over  2500  entities.  Figures  8  and  9  provide  bandwidth  information  that  was 
collected  on  the  DIS  and  HLA  networks  at  AMRDEC. 
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Figure  6.  Latency  from  AMRDEC  to  Sites 


Figure  7.  Throughput  Tests  from  the  Sites 
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Figure  8.  Average  Bandwidth  Utilization 
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Figure  9.  Maximum  Bandwidth  Utilization 


IV.  RECOMMENDATIONS 

The  network  built  for  the  RDE  IstApp  provided  a  stable  environment  that  contributed  to  the 
success  of  the  “Runs  for  Record.”  Success  for  future  distributed  exercises  lies  in  careful 
planning  and  preparations.  Thus,  it  is  important  to  address  ways  to  improve  the  LAN  and 
MAN/WAN  architectures. 

A.  LAN 

It  is  important  that  ATM  QoS  services  be  supported  in  the  future.  To  accomplish  this, 
it  is  recommended  that  network  equipment  that  can  fully  support  this  be  identified  and  installed 
at  each  participating  site.  Compatibility  of  equipment  throughout  the  network  is  extremely 
important.  It  is  recommended  that  a  test  laboratory  be  set  up  and  equipment  tested  prior  to  being 
recommended  and  acquired.  It  is  also  recommended  that  a  network  toolset,  that  could  address 
performance  and  management  issues,  be  identified  and  installed  at  each  site. 

B.  MAN/WAN 

The  architecture  at  the  MAN  layer  at  the  sites  is  still  problematic.  A  reduction  in  the 
number  of  switches,  routers,  firewalls,  etc.  is  necessary.  Ideally,  a  LAN  would  be  only  one  hop 
away  from  an  SDP’s  demarc  point. 

It  is  also  recommended  that,  to  support  classified  experiments  in  the  future,  sites  join 
the  SDREN.  A  discussion  of  the  effort  needed  to  acquire  an  MOA  to  support  this  application, 
was  beyond  the  scope  of  this  report.  However,  it  proved  to  be  a  very  difficult  process.  The 
SDREN  affords  many  services  to  include  support  for  Fastlane  keying  material,  Fastlane  technical 
support,  and  network  management  support  to  name  but  a  few.  There  is  a  12-step  connection 
approval  process  required  prior  to  joining  the  SDREN,  and  sites  agree  to  fund  a  security  team  to 
visit  their  site  for  one  week  each  year  to  validate  security. 
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V.  CONCLUSION 


The  network  used  to  support  RDE  1  stApp  was  planned  and  implemented  over  a  very  short 
five-month  period.  The  sites  involved  are  extremely  grateful  and  appreciative  for  the  level  of 
support  that  the  DREN  provided  to  this  effort.  As  active  members  of  the  team,  their  efforts 
contributed  significantly  to  the  success  of  the  RDE  1  stApp  effort.  This  effort  was  ongoing  as  the 
DREN  was  still  in  transition  from  an  American  Telephone  and  Telegraph  company  (AT&T)  to  a 
WorldCom  backbone. 

The  future  success  of  distributed  simulation  efforts  are  going  to  be  dependent  on  research, 
and  the  provision  of  resources  to  support  an  integrated  network  architecture. 
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